1. Field of the Invention
Embodiments of the present invention relate to electronic aids for the blind (or individuals with low vision) and prosthetic vision. Thermal imaging is disclosed as an alternative to or enhancement to visible imaging, resulting in an improved interpretation of an observed scene.
2. Description of Related Art
It is estimated that 120,000 Americans are totally blind and several million Americans experience significant vision loss. Devices developed to aid and assist blind and low-vision persons generally fall into the following categories: sensory substitution, sensory augmentation, visual prostheses, and low-vision enhancement devices. Sensory substitution and visual prosthetic devices are applicable to blind and low-vision persons. Sensory augmentation and low-vision enhancement devices are only applicable to persons that still have some amount, however limited, of operative vision.
Sensory substitution is a method that utilizes sensors to feed an annunciation device that stimulates an alternative sense. In a typical configuration, a camera is connected to a device that generates audio cues or a device that stimulates the sense of touch (i.e., haptic). Haptic devices can be electrostatic stimulation arrays, electromechanical pin arrays, or an array of electromechanical vibrators. Sensory augmentation methods operate similar to sensory substitution but assume that some amount of vision is still present in the person.
Visual prosthetic methods attempt to replace or bypass the specific inoperative biological functions with synthetic devices or implants in order to restore the overall visual function. One type of visual prosthetic method uses electronic sensors to direct the image of a retinal stimulating implant.
Low-vision enhancement devices use electronics and/or optics to magnify, enhance, or warp a visual scene in a manner that maximizes perception for persons who have lost aspects of their visual function.
Traditionally, a problem that devices based on each of these methods have in common has been the low effective bandwidth of the information channel. Audio, haptic, and even retinal stimulation implant devices can only provide a small portion (<1%) of the information bandwidth of a well-functioning eye.
Traditionally, each of these assistive methods have utilized visible light video cameras as the primary input sensor. There has been recent research and experimentation with the use of electrotactile arrays (also called haptic displays) to communicate visual images (normal light) to the blind. Fundamentally, it is a very complex problem to translate an image in a real world environment into meaningful tactile data. Image recognition techniques to identify edges of different separate objects must be used. This level of processing is not very practical in wearable applications. The use of thermal imaging has not been explored because in the past cryogenically cooled thermal cameras have been too expensive and too bulky to be practical.
Thermal Imaging
The electromagnetic spectrum includes ultraviolet (wavelengths from 0.1 to 0.4 micrometers), visible (from 0.4 to about 0.75 micrometers), near-infrared (from 0.75 to 1.2 micrometers), mid-infrared (from 2 to 5 micrometers) and far-infrared (from 8 to 14 micrometers). All materials at temperatures above zero degrees Kelvin emit infrared radiation. Most naturally occurring terrestrial objects have peak infrared emissions in the 8 to 14 micrometer range (far-infrared). Thermal imaging is done in the 8 to 14 micron range. In this range glass is opaque. Thermal imaging is done with lenses made of material such as germanium. Germanium is opaque to visible light. It is not presently considered possible for a visible light imager to share the same optics as a thermal imager.
Early thermal imaging systems developed in the 1970s and 1980s were unwieldy and did not lend themselves well to many applications. Physically large and technically complex, they required expensive liquid nitrogen or similar cryogenic cooling systems. Thermal imaging systems have been slow in delivering greater operational flexibility because of the cost, size, and weight of the cryogenic cooling components used in prior generations of high-performance IR sensors, and because of the size and power consumption of the supporting electronics.
In the early 1990s, a revolutionary suite of imaging radiation sensors was developed (see, e.g., U.S. Pat. Nos. RE036615, 6,114,697, 5,554,849, and 5,834,776, all of which are incorporated herein by reference). These sensors were revolutionary because they are mass-producible from materials such as low-cost silicon, and they operate well at room temperatures (hence termed “uncooled”).
Uncooled IR sensors, such as of the microbolometer type, typically include arrays of microscopic bridge-like structures micromachined from silicon. Given the extremely low mass of the microbridge structures (typically on the order of a nanogram), they respond to very low radiation levels. Accurate measurements of microbridge temperature changes are used to quantify incident thermal radiation. Common methods for measuring microbridge temperatures include the use of thin-film thermocouples to generate a thermoelectric (TE) signal, or the use of thin-film resistors that undergo resistance changes according to temperature.
The basic operating principle of an uncooled silicon IR detector is as follows. Thermal energy in the 8 to 14 micron wavelength emitted from the target object is focused onto an extremely low mass microstructure. The incident energy is absorbed by the microstructure and causes an increase in the temperature of the bulk of the material. This temperature rise can be exactly correlated to the temperature at the optically corresponding point on the target.
Known uncooled IR imaging sensors include arrays of microscopic (typically 0.05 mm wide and 0.001 mm thick) bridge-like structures “micromachined” into silicon wafers by photolithographic processes similar to those used to make microprocessors. Calculation of the heating of microbolometers produced by focused IR radiation can be made using the well-known physical laws of radiation, and such microbolometers can measure temperature changes in a remote object with sensitivity well below 0.1° C.
For best sensitivity, microbolometer arrays should operate in an air pressure of 50 mTorr or less in the vicinity of the pixels, to eliminate thermal loss from the pixel to the air. To minimize size and weight and production costs, a process disclosed in U.S. Pat. No. 5,895,233, incorporated herein by reference, discloses a device allowing the completed array to have an infrared-transparent silicon top cap hermetically attached, to form an all-silicon integrated vacuum package (IVP). This technique allows a microbolometer imaging array to have small dimensions. Known microbolometer packages require a vacuum-sealed package around the outside of the microbolometer, resulting in larger diameters. Arrays are typically close-packed across the wafer, with a very small spacing to allow wafer sawing to separate completed arrays.
Because the sensors are fabricated using silicon photolithography, it is cost-effective to fabricate large one-dimensional (1D) and two-dimensional (2D) arrays complete with monolithic silicon readout electronics if required for a particular application. Two-dimensional arrays of IR sensors may be used with an IR-transmitting lens to produce a 2D temperature map of a target, analogous to the way a visible camera produces a two-dimensional image of a target.
Other methods have also been developed to construct arrays of infrared radiation detectors, including the use of pyroelectric detector elements, p-n junction devices, microcantilevers, or photoconductive or photovoltaic bandgap materials.
Blind and Low Vision
An application of infrared temperature measurement called the “People Sensor” is disclosed in Ram, S., et al., “The People Sensor: A Mobility Aid for the Visually Impaired”, IEEE, 1998, pp. 166-167. The People Sensor combines an ultrasonic distance sensor with a pyroelectric IR sensor, to determine if an object that was identified by the ultrasonic sensor was animate (human) or inanimate (non-human). Only a single point measurement was taken.
There has been a history of work involving people attempting to use visible light images as an input to some type of haptic interface to a person. In the 1960s, Dr. James Bliss and his colleagues developed the Optacon, a tactile sensory substitution reading aid for the blind. The Optacon consists of a 6×24 element photodiode (light-sensitive) array that is mapped onto a 6×24 matrix of vibrating reeds, where the user places his finger to sense the image picked up by the light-sensing array. Subjects trained on this device were able to achieve reading rates of 60 words per minute.
Also in the 1960's, at the Smith-Kettlewell Institute of Visual Science, Dr. Bach-y-Rita and his colleagues developed a large electromechanical array of 400 points mounted in a chair which would transmit patterns of vibration onto the back of a person sitting in the chair. The patterns of vibration were dictated by the images sensed by a television camera under the control of the person in the chair. If the camera were directed towards a white vertical line on a black background for instance, the person would feel a vertical line on their back. If the camera were moved to the right, they would feel the line move correspondingly on their back. Although such a system did not have anywhere near the ability of the human eye to gather visual information, it showed that the brain was indeed capable of perceiving visual information through the skin.
Because electromechanical components are noisy, costly, consume a lot of electrical power, and have very limited reliability, efforts were made to develop electrical tactile stimulators. While these efforts were able to overcome many of the problems associated with electromechanical stimulators, new problems with comfort of sensation and skin irritability, and oftentimes even skin burns surfaced. These problems would have to be overcome before electrical stimulation could be practically used in tactile feedback applications. In the early 1990's ways to develop a multi-channel electrotactile system that would be able to stimulate the skin in a safe and comfortable manner began to be developed. This work was led by Kurt Kaczmarek at the University of Wisconsin. A recent example of a haptic interface of visible imaging is U.S. Pat. No. 6,055,048, which is incorporated herein by reference. In this patent, an optical sensor is described that operates in the far ultraviolet, visible and near infrared (up to 1 micron wavelength). A person who also has a haptic interface to sense patterns created from the processed optical sensor wears the sensor. The patent states that a microprocessor on the person has algorithms to recognize common shapes such as traffic lights, trees, cars, doorways and so forth. In an actual system this type of processing would be very difficult and require extremely high processing power.
There have been other applications of infrared light used as part of an aid or assist device for blind and low-vision persons. These applications typically use infrared for signaling or orientation. This is similar to applications of infrared for remote controls and uses near infrared wavelengths (0.75 to 1.2 microns). Near infrared has been used as an alarm system for blind and visually impaired persons (see U.S. Pat. Nos. 5,838,238 and 5,933,082, which are incorporated herein by reference). Installed devices create beams of near infrared light to warn a person when they are approaching a hazardous area. This is an example of near infrared being used to signal and provide orientation. Another example is given in a paper by Ertan et al. from MIT. In this example, ceiling-mounted infrared transceivers are used as a signal to a person to identify which room they are in.
Near infrared light has been used to provide distance information to a blind or low-vision person (see U.S. Pat. Nos. 5,487,669, 6,298,010, and 5,807,111, incorporated herein by reference). In these applications the blind person would use one or more IR distance measurement device to provide them warnings or orientation to obstacles and barriers. In these applications a laser or other IR light source is reflected from the obstacle or barrier and sensed by a device used by the blind or low vision person.